title: Plotting data from large file in Python
body: <p>I have a large file (2 GB). I want to plot the data in it in <code>scatterplot</code>. The data has following format in the file.</p>
  <pre><code>day block1  block2  block3  .....
  1   34.89   88.90   67.89   .....
  2   77.890  33.56   76.98   .....
  3   67.12   67.89   55.89   .....
  ... .....   .....   .....   .....
  </code></pre>
  <p><strong>pltData</strong> will be the list of average of the column that is</p>
  <pre><code>pltData = [avg_block1, avg_block2, avg_block3, .....]
  </code></pre>
  <p><strong>pltX</strong> and <strong>pltY</strong> are finite lists. For plotting the data from list named as <strong>pltData</strong>, I'm using following code:</p>
  <pre><code>from matplotlib.colors import ListedColormap, LinearSegmentedColormap
  import matplotlib.pyplot as plt
  import matplotlib.cm as cm
  
  FIGURE = plt.Figure()
  SUBPLOT1 = FIGURE.add_subplot(121)
  SUBPLOT1.set_xlabel('x distance')
  SUBPLOT1.set_ylabel('y distance')
  data1 = {'x-distance' : pltX , 'y_distance' : pltY}
  df3 = DataFrame(data1, columns=['x-distance','y_distance'])
  plot1= SUBPLOT1.scatter(df3['x-distance'],df3['y_distance'], marker='o', s=15, linewidths=0.1, c=pltData, cmap='rainbow', vmin=min(pltData), vmax=max(pltData))
  FIGURE.colorbar(plot1, ax=SUBPLOT1)
  </code></pre>
  <p>However, for file as large as 2GB, creating list <strong>pltData</strong> is impossible because number of rows and columns are large. Can someone guide me to the way I can plot the data?</p>