title: Which technology should be used for serving large number of static files?
body: <p>My main aim is to serve large number of XML files ( > 1bn each &lt;1kb) via web server. Files can be considered as staic as those will be modified by external code, in relatively very low frequency (about 50k updates per day). Files will be requested in high frequency (>30 req/sec).</p>
  
  <p>Current suggestion from my team is to create a dedicated Java application to implement HTTP protocal and use memcached to speed up the thing, keeping all file data in RDBMS and getting rid of file system.</p>
  
  <p>On other hand, I think, a tweaked Apache Web Server or lighttpd should be enough. Caching can be left to OS or web server's defalt caching. There is no point in keeping data in DB if the same output is required and only queried based on file name. Not sure how memcached will work here. Also updating external cache (memcached) while updating file via external code will add complexity.</p>
  
  <p>Also other question, if I choose to use files is is possible to store those in directory like \a\b\c\d.xml and access via abcd.xml? Or should I put all 1bn files in single directory (Not sure OS will allow it or not).</p>
  
  <p>This is NOT a website, but for an application API in closed network so Cloud/CDN is of no use.</p>
  
  <p>I am planning to use CentOS + Apache/lighttpd. Suggest any alternative and best possible solution.</p>
  
  <p><a href="http://blogs.law.harvard.edu/philg/2007/01/05/best-web-server-program-for-a-lot-of-static-files/" rel="nofollow noreferrer">This</a> is the only public note found on such topic, and it is little old too.</p>