title: Large data with pivot table using Pandas
body: <p>I’m currently using Postgres database to store survey answers.</p>
  
  <p>My problem I’m facing is that I need to generate pivot table from Postgres database.</p>
  
  <p>When the dataset is small, it’s easy to just read whole data set and use Pandas to produce the pivot table.</p>
  
  <p>However, my current database now has around 500k rows, and it’s increasing around 1000 rows per day. Reading whole dataset is not effective anymore.</p>
  
  <p>My question is that do I need to use HDFS to store data on disk and supply it to Pandas to do pivoting?</p>
  
  <p>My customers need to view pivot table output nearly real time. Do we have any way to solve it?</p>
  
  <p>My theory is that I’ll create pivot table output of 500k rows and store the output somewhere, then when new data gets saved into database, I’ll only need to merge the new data with existing pivot table. I’m not quite sure if Pandas supports this way, or it needs a full dataset to do pivoting?</p>