title: Cache huge data in-memory
body: <p>I am looking for an in-memory cache solution which can handle big data (&lt;5GB). For a user inputted search term, the database (elasticsearch) will return a large amount of data which the tool will analyze and show via different webpages of the tool. Now my problem is that I want to cache this big data temporarily till the user session gets over so that I don't have to fetch it again from elasticsearch every time the user opens a new page. It will have to be in-memory because disk based will take over a minute which would be very slow.</p>
  
  <p>I initially thought memcached but it has a max limit of 128MB. After reading quite a bit, Redis seems suitable but it is unclear to me whether a bunch of Redis nodes can work in tandem or not. Is it possible to set up a pool of many Redis nodes so that a suitable node will be automatically chosen on <code>SET</code> and the data returned upon <code>GET</code> without me having to specify the node?</p>
  
  <p><strong>TL;DR</strong></p>
  
  <ul>
  <li><strong>Problem</strong>: Cache big data (&lt;5GB) in an in-memory cache </li>
  <li><strong>Possible solution</strong>: Redis</li>
  <li><strong>Question</strong>: Can I pool a bunch of Redis nodes so that I can fetch a key stored in any of them without specifying a particular node. I don't need to distribute my data since data for a single user will fit into the RAM of a single node.</li>
  </ul>