title: Python/Pandas how to store forex tick data for analysis
body: <p><sup>I am new to programming, Python and Pandas so hopefully this isn't a silly question.</sup><br><br>I have downloaded some FOREX data from <a href="https://www.truefx.com/?page=downloads" rel="nofollow"><strong>here</strong></a>.  A month's worth of data is around 50mil lines in CSV format for all pairs.<br><br>I would like to eventually be able to test a strategy across multiple time frames and instruments.</p>
  
  <p>Here is the code I'm using:</p>
  
  <pre><code>file_address = '/Users/Oliver/PyCharm/FX_app/test_data/EURUSD_test.csv'
  
  df = pd.read_csv( file_address,
                    names       = ['Symbol', 'Date_Time', 'Bid', 'Ask'],
                    index_col   = 1,
                    parse_dates = True,
                    converters  = { 'Date_Time': convert_string_to_datetime }
                    )
  
  #                                                               a non-PEP8 format
  #                                                               didactic purpose
  </code></pre>
  
  <p>On anything but a truncated test file this reading in process takes a long time.</p>
  
  <ul>
  <li>Is there a way I should be storing the data so that Pandas can read the files much faster?</li>
  <li>Is there a limit to the size of the data that Pandas can reasonably handle?  </li>
  </ul>
  
  <p>Any help would be greatly appreciated.</p>